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ABSTRACT 



The Feasibility of developing an index to represent 
the degree of consistency with which a student responds to a set of 
items in a homogeneous scale is investigated. The dispersion of 
responses to a set of items by a group of subjects has typically been 
measured by the standard deviation. Here, the interest is in the 
dispersion of responses from the same subject to a set of items- The 
Ohio Vocational Interest Survey was used to gather students* 
responses to 11 items (job-activity statements) in terms of a 
5-option scale. The investigation consisted of 3 phases: (7) a scale 

clarity index (SCI) was devised; (2) all possible response patterns 
were defined and the clarity of each was rated in terms of degree of 
consistency; and (3) the extent to which the deviation raw score^ the 
standard deviation, mean deviation, and a centrality index would 
predict the scale clarity rating developed in the second phase was 
investigated. A critical aspect of this study is the judges* ratings 
of the clarity inherent in each pattern. The SCI provides additional 
information as to the nature of a subject’s responses^ is useful in 
the interpretation of raw scores, especially those based on 
individual item responses, and has great promise in testing and 
measurement practice. (TA) 
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Background 

Individual variations in responding to the statements within a scale 
consisting of a set of homogeneous items appear to have been overlooked 
as individual idios 3 nicrncies or normal measurement problems. However, 
information about an individual’s consistency in responding to a set 
of homogeneous statements can give added meaning to the interpretation of 
his responses to the items in a scale. How can this information be 
quantified and reported in a practical manner? 

Purpose 

The purpose of this investigation is to examine the feasibility of 
developing an index to represent the degree of consistency with which a 
student responds to a set of items in a homogeneous scale. The dispersion 
of responses to a set of items by a group of subjects has typically been 
measured by the standard deviation. This study is interested in the 
dispersion of responses from the same subject to a set of items. 

Method 



The Ohio Vocational Interest Survey (OVIS) (D* Costa et al . 1969) is 
an instrument designed to help high-school students to explore the world 
of work. OVIS consists of 24 scales, each comprising 11 job-activity 
statements which were judged to be representative of a particular area 
within the world of work. The OVIS scales were initially derived on an 
a priori basis from the Cubistic Model of Vocational Interests (D* Costa 
and Winefordner, 1969) • The items comprising the scales were selected 
and defined empirically to provide homogeneity in the scales. 
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In OVIS, students are asked to respond to each job-activity state- 
ment in terms of the five-option scale, **Like very much," "Like," "Neutral," 
"Dislike," and "Dislike very much," In computing a score for each scale, 
a value of 1 is assigned to "Dislike very much," 2 to "Dislike," 3 to 
"Neutral," 4 to "Like," and 5 to "Like very much," A scale score is de- 
fined as the sum of a student *s responses to the 11 items in the scale and 
indicates the strength of his liking for the job activities that make up 
that scale, A scale score can range from 11 to 55- Thus, to obtain a 
scale score of 49, a student must indicate that he "likes very much" most 
of the activities making up that scale. His average response score of 4.5 
reflects the level of his responses. However, for scale scores in the 
middle range of 24 to 42, the average response score may not accurately 
reflect the variations of a student's interests within the scale. For 
example, a student with a scale score of 33 could be uniformly neutral to 
the activities making up the scale, or his score could represent the sum 
of high and low weights reflecting likes as well as dislikes. Since the 
11 items were selected to yield a homogeneous scale, the degree of con- 
sistency with which a student responds to the items can be considered to 
be an index of scale clarity that the individual has attained within him- 
self. 

The first phase of this investigation considered a scale clarity 
index (SCI) which was defined as a linear function of the standard devia- 
tion of the individual's responses to the 11 items in a scale. Of course, 
many different patterns of response are possible. For example, one pattern 
of responses is: 

Response Type D d 

Weight (1) (2) 

Pattern of Response 0 0 




2 



N 

(3) 

0 



1 

(4) 

2 



L 

(5) 

9 
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Here, the mean response weight is 4.B (scale score of 53) and the standard 
deviation of responses is 0.4, The scale clarity score was defined as 
25 - 10(s.d.)* The interpretation of this score was done on a normative 
basis, that is, with reference to the frequency distribution of scale 
clarity scores obtained by a criterion group. A decision was made to 
label scores at or above the third quartile (Q 3 ) as highly consistent, 
those between and Q 3 as fairly consistent, and those at or below 
as inconsistent. 

There were several limitations noted with respect to this approach: 

1. It was assumed that the standard deviation, or a function 
of it, alone is an accurate measure of scale clarity in- 
herent in a response pattern. 

2. That it is appropriate to interpret the scale clarity 
score for an individual in reference to a norm group. 

An index of the subject’s level of consistency thus 
becomes dependent upon the consistency of others. 

3. The normative approach, which was applied independently 
for each scale, implies that it is appropriate to use 
different standards to judge clarity in different scales. 

Thus, a student with the same response pattern on two 
different scales and showing the same consistency with- 
in himself, could conceivably be labelled "highly con- 
sistent" on one scale and "fairly consistent" on the 
other . 

The second phase of this investigation proceeded on the premise that 
the response pattern is the key to the clarity question. Furthermore, it 
was believed that scale clarity can be best defined in an absolute and 
uniform manner. It was decided to: 1) Define all possible patterns of 

response to the 11 iter^is in a scale, 2 ) Use experienced judges to rate 
the clarity perceived in each of the possible patterns of response as 
"Highly Consistent," "Fairly Consistent," or "Inconsistent," and 3) Compute 
the raw score and the standard deviation or variance corresponding to each 




o 

o 



pattern. 



To determine all possible patterns of response, it was recognized 
that there are 11 items in a scale, that the subject responds independently 
to the 11 items, and that the subject can respond to each item in any one 
of five different ways. The simulation took note of the fact that in any 
one scale, a response could be made not at all, or as many as 11 times. 

It was found that there arc 1365 unique response patterns possible. A 
complete listing of these 1365 patterns and their standard deviations was 
obtained by a computer simulation program. But since it was found that 
the 1365 patterns yielded only 116 unique standard deviations, the use of 
the standard deviations alone as a measure of scale clarity was deemed 
inappropriate. 

Since a particular standard deviation could result from apparently 
different patterns of consistency, each of the 1365 patterns was examined 
to identify the clarity inherent in each. A panel of three judges was 
instructed in the technique and a set of ratings from each judge was ob- 
tained. Clarity was looked at in terms of the homogeneity of the responses 
i*e., the number of identical responses, the number of dissimilar responses 
and the location of the cluster of responses on the 5-point response 
continuum. 

It was noted that the patterns occurred symmetrically around the raw 
score of 33. Extreme scores, such as 11-17 and 49-55, had patterns that 
reflected highly consistent responses; the inconsistent response patterns 
occurred generally in the middle range of scores. As expected, most of 
the greater standard deviations occurred in the middle range of scores. 
Therefore, each pattern's raw score and standard deviation were examined 
along with the judges* ratings in an attempt to produce meaningful scale 
clarity Indexes. 



The second phase ended with the development of a table to permit one 
to read off a scale clarity index corresponding to a particular raw score 
and standard deviation. 

The third phase of this study is an investigation of the extent to 
which the deviation raw score (defined as the absolute value of the raw 
score minus 33), the standard deviation, mean deviation, and a centrality 
index (defined as a mean deviation around the ^Neutral” response) would 
predict the scale clarity rating developed in the second phase. 

Results and Conclusions 

Table 1 presents the intercorrelations among the four predictor 
variables, i.e., deviation raw score, standard deviation, mean deviation, 
and centrality index, and the criterion variable, namely the scale clarity 
index determined under Phase 2. 



Table 1. Intercorrelations among the deviation raw score, standard devia- 
tion, mean deviation, centrality index, and scale clarity index for every 
possible pattern of response for the OVIS scales. 





DRS 


SD 


MD 


Cl 


SCI 


Deviation Raw Score 


1.00 


-.31 


-.32 


.43 


.67 


Standard Deviation 




1.00 


.96 


.63 


-.77 


Mean Deviation 






1.00 


.61 


-.76 


Centrality Index 








1.00 


-.23 


Scale Clarity Index 










1.00 






It is interesting to note that the deviation raw score, the mean deviation, 
and the standard deviation each predicts the scale clarity index determined 
by the panel of judges at a relatively high level. 

However, since certain of the predictor variables are highly correlated, 
the second stage in this phase of the investigation was concerned with the 
determination of measures as unique predictors of scale clarity. For this 



reason, a stepwise regression analysis was undertaken to determine those 



measures yielding the greatest multiple correlation in the prediction of 
criterion. Table 2 presents the results of this analysis. 
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Table 


2. Stepwi 


se regression analysis of 


the deviation raw score, 


standard 


deviation, mean 
clarity index. 


deviation, and centrality 


index as predictors of 


the scale 






Predictor 


R 


r2 


Step 


1 


Standard Deviation 


.766 


.586 


Step 


2 


Standard Deviation 
Deviation Raw Score 


.890 


.793 






Standard Deviation 






Step 


3 


Deviation Raw Score 
Centrality Index 


.897 


.805 



The results of this analysis are not surprising for the response patterns 
judged to be inconsistent occurred generally in the middle range of scores 
where the standard deviations are also generally greater. 

However, a critical aspect of this study is still the judges' ratings 
of the clarity inherent in each pattern. How to improve the accuracy of 
this criterion still remains a problem. The approach in this outgoing 
phase of this study will be one of successive approximations. Thus, we 
hope we can improve our scale clarity indexes and, in turn, go on to 
quantify them to reflect the within-subject variability in a set of homo- 
geneous items in a scale. 

The scale clarity index provides additional information as to the 
nature of a subject's responses and is useful in the interpretation of 
raw scores, especially those based on individual item responses, and has 
great promise in testing and measurement practice. 

The scale clarity index also provides an interesting question for 
statisticians. How good a measure of dispersion is the standard deviation? 
To what extent does the mean deviation supplement the information provided 
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by the standard deviation. Can some other index, such as the centrality 
index, be developed, which might add to the definition of the clarity 
in a response pattern? 

This study does not have the answers to all these questions. But 
its value lies in the fact that it can be useful in measurement practice. 
Use of the concept in relation to the Ohio Vocational Interest Survey 
suggests that it is indeed useful. 
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